Modern operating systems are actually a modular collection of building blocks rather than one monolithic object. This form of architecture enables an Operation System (OS) manufacturer to build and distribute new facilities (or new versions for existing facilities) with relative ease. It also enables third party programmers to add new capabilities to a basic operating system without accessing its source code, by means of well-defined extension interfaces.
In particular, the Input/Output (I/O) architecture of a conventional operating system is multi-layered, scaleable and extensible, i.e., each packet of I/O data travels along a chain of layers, wherein the layers are organized so that applications that are being executed by the OS at the system of a user are divided into separate functional components that interact in some sequential and hierarchical way, with each layer in the chain usually having an interface only to the layer above it and the layer below it. Some of the layers are those provided with the original software package of the OS, some are updates, others are additional layers not present in the original software package, and finally, some are actually built by third party suppliers (including original parts of the original software package that were subcontracted).
Operating systems usually make a distinction between a privileged mode and non-privileged mode regarding the ability of a process to call ‘privileged’ services. The notion of multi-layered architecture and underlying extension mechanisms apply to both modes, but the implementation may differ significantly.
A Privileged mode (so-called “Kernel-mode”) is the essential core of any OS, which provides basic services for other parts of the OS. Typically, the Kernel-mode is the part of the OS that resides in the memory of the computer at all times during its operation, and provides basic services. It is the part of the OS which is closest to the machine level and may directly activate the hardware of such a computerized system, or interface with another software layer which drives a hardware. Due to performance considerations, kernel-mode processes typically share the system's physical memory space without an extra mapping of their non-privileged mode relatives. A kernel-mode process can be seen as a server to many non-privileged mode processes, which is vulnerable to possible low-level breaches. A process of a non-privileged mode (so-called “user-mode”) can call system services that are not privileged.
In particular, user-mode multi-layer extension mechanisms are quite vulnerable to memory-space breaches. These enabling mechanisms are relatively well documented, and a dedicated programmer having access to the interfaces of these extension mechanisms is generally able to implement them in quite a short time. Some well known books in the field of general Operating Systems, particularly Windows™ OS provide relevant information regarding this subject.
From now on, and unless otherwise stated, the following text will refer to user-mode.
The building blocks of standard applications, such as a word processor, an Internet browser etc., are code modules, usually divided into program modules and/or into Shared Code Resources (SCRs). Examples for such SCRs are the Dynamic Link Library (DLL), which are included in the Windows OS of Microsoft. Furthermore, each application may use several SCRs on the same session. Generally, SCRs are grouped in stacks, wherein each stack contains several SCRs, sometimes a dozen or more SCRs are grouped together in one stack. The SCRs are organized in each stack in a chain-like manner. Normally, whenever a service from a specific SCR is requested by a user application, the request travels along the whole relevant stack, however, the user application has no clue about the specific SCRs that actually serve it along the way.
Moreover, when an extension is needed to one of the OS services, for example, encrypting certain I/O data packets, an insertion of an SCR into the relevant stack chain should do. Of course that SCR has to comply with a given interface and be good mannered, the least it should do is to dispatch incoming calls to the next SCR in the chain.
In a typical case, an extender, which might be, for example, a specific process, requests from the OS to insert an SCR into a specific extensible chain. If all goes well, the SCR is inserted as a new “layer”, and starts receiving relevant calls as if it was an original part of the stack, and of the OS. From this point on, until this new SCR is appropriately removed from the chain, the newly installed SCR is mapped into the address space of any application that happen to use that relevant stack.
Due to resource-economy considerations, a reasonable multitasking operating system would load just a single copy of a given SCR into the physical memory, and then map it to the virtual address space of each process that might need it. More particularly, each ‘instance’ of the SCR is mapped to the appropriate process context. Unfortunately, there is more than one way to share memory between the SCR's ‘instances’.
Combining the aforementioned factors that compromise the requirement for separation between memory spaces of different processes, there is an opportunity for offenders to abuse the inherent mechanisms of the operating system. In fact, this provides a possible way for one process to break into the memory space of another process.
An offender that has managed to break into the memory space of another process has a choice of options. Amongst other threats, the offender may read or manipulate I/O, it might change the behavior of the invaded application, or it may send information from one process to another process.
One of the most serious aspects of memory-space breaches is the ability of the offender to take the identity of the invaded process. This makes life harder for auditing tools and intrusion-detection systems, and makes the search for accountability more difficult.
In general, memory-space breaching seems as an appropriate technique for an invader, whose goal is not mere vandalizing of an invaded site. Sophisticated contamination of the victim's valued information resources would be a possible goal. Altering the behavior of unaware information-security systems through their user-mode components is another goal, so is eavesdropping or stealing information, to mention just a few.
For example, a ubiquitous OS like MS Windows™ (Microsoft Corporation, USA) gives a program the ability to order the OS to extend, on the fly, SCR stacks, wherein each SCR provides a particular level of functionality. In many cases this goes on without alerting the user of this OS. The extension is done by adding at least one SCR to the chain, wherein this SCR may serve more than one application or process concurrently.
There are several mechanisms in the OS that might be extended by additional SCRs. The following are examples for some well known in the art of such mechanisms:    (i) windows-messages that may be hooked;    (ii) video and audio Compressor/De-Compressor (codecs);    (iii) Windows Open Services Architecture (WOSA) stacks, which is a collective term for a variety of programming interfaces from Microsoft designed to provide application interoperability across the Windows environment. An example for WOSA is the Windows Socket (Winsock), which is a Windows interface to a communications protocol over the Internet; and    (iv) There are more WOSA mechanisms like ODBC and MAPI. Furthermore, there is the infrastructure for the ‘Component Services’ of Windows. This list is by no means a complete list of all the vulnerable service chains in a modern OS, but only a list of examples.
The art has not yet provided satisfactory protection means for detecting and/or preventing such inter-process memory breaches in multitasking OS.
It is an object of the present invention to improve the security in multi-users and multitasking systems.
It is another object of the present invention to provide a method and system for detecting an illegal action of penetrating a memory space of one process by another process.
It is further object of the present invention to provide a method and system for detecting a process that initiates such penetrating action.
It is a still further object of the present invention to provide a method and system for freezing the action of the invader and/or the invaded processes, and alerting on such illegal action.
Other objects and advantages of the invention will become apparent as the description proceeds.